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EVALUATION OF A VERTICAL -SCALE, FIXED-INDEX 

INSTRUMENT DISPLAY PANEL FOR THE X-15 AIRPLANE 

By Lee E. Lytton 
Flight Research Center 


SUMMARY 


A comparative evaluation was performed on an analog simulator to compare pilot 
performance when using the operational X-15 instrument panel and a panel incorpo- 
rating vertical -sc ale, fixed-index flight instruments. The purpose of the evaluation 
was to provide experimental evidence to complement pilot opinion concerning the ac- 
ceptability of the vertical -scale panel for use in the X-15 airplane. This evidence was 
obtained in the form of a wide variety of performance measures for 16 subjects, for 
two different representative mission profiles, and over three trials or runs for each 
profile. The data were subjected to both parametric and nonparametric statistical 
analysis. 

Performance differences between panels were largely either insignificant or were 
masked by variability within subjects. Pilot opinion, as expressed in questionnaires, 
generally supported this result, in that performance levels for the two panels were 
judged to be equivalent. Pilot reports, after five X-15 flights, indicate increasing 
acceptance of the vertical -scale panel as well as operational substantiation of the 
results of the simulator evaluation. 

The need for better indices of performance in displays evaluation seems clear, 
considering the lack of evident distinction where it is intuitively expected and the fact 
that the techniques used in this investigation were current. 


INTRODUCTION 


The vertical -sc ale instrument display panel now being flight tested in the X-15 air- 
plane represents a major departure from traditional instrument panel design. There is 
a basic difference in design philosophy between the individual indicators on the vertical- 
scale panel and those on the standard model, as well as considerable difference in ar- 
rangement of the instruments. The panel, which is the result of work carried out under 
USAF Advance Technology Program 667A (ref. 1), is referred to as the advanced con- 
trol display subsystem (ACDS) panel. 

The design philosophy followed in the ACDS panel established a common reference 
or "read-line" for the primary flight instruments in order to reduce both the time 
spent in scanning instruments and the complexity of the scan pattern. A considerable 
gain in display sensitivity was obtained by using tape scales up to 40 inches long. 


Direction of movement of the tape scales was chosen for each parameter to represent 
most closely the naturally expected result of a particular control action. The ACDS 
design philosophy is discussed further in reference 1. 

Because of the obvious differences between the standard and the ACDS panels and 
because of evidence of pilot nonacceptance of vertical -scale fixed-index (tape) 
instruments, an objective evaluation was conducted at the NASA Flight Research Center 
to determine the suitability of the ACDS panel for flight. The comparison attempted 
was between two specific, complete panels and not between the concept of vertical- 
scale and circular-dial instruments. Prior to this study, these display systems had 
been evaluated almost exclusively in terms of pilot opinion, and the few objective 
measures taken had concentrated largely on end-point conditions (i. e. , merely on peak 
altitude achieved, for example, versus the planned value). 

With the contracted aid of the Life Sciences Group of Douglas Aircraft Company, 
an experiment was designed that would enable a more objective comparative evaluation 
to be made of the panels and would provide results that would be significant in a 
statistical sense. The experimental design recommended by Douglas called for selec- 
tion of a sizable group of reasonably qualified subjects and the choice of typical ap- 
propriate X-15 mission profiles. Required also was a set of dependent variables that 
would serve both as accurately representative and clearly discriminant measures of 
performance differences and would lend themselves readily to analysis by statistical 
techniques. Douglas’ participation was based on experience gained during an earlier 
study (ref. 2) involving auditory flight displays as applied to the X-15, inasmuch as 
techniques employed therein were directly applicable to the display-panel evaluation. 

The purpose of this paper is to describe the design and conduct of the experiment, 
to report the results of the comparative evaluation, and to point out the need that be- 
came apparent for development of better indices of performance in displays evaluation. 


SYMBOLS 


The units used for the physical quantities in this paper are given in the U. S. 
Customary Units and, where applicable, in the International System of Units (SI). 
Factors relating the two systems are presented in reference 3. 

g acceleration due to earth’s gravity, 32.2 feet/second^ (9. 8 meters/second^) 

h inertial altitude, feet (meters) 

ti altitude rate, feet per second (meters per second) 

q dynamic pressure, pounds per foot^ (newtons per meter ) 

t time, seconds 

V total velocity, feet per second (meters per second) 

a angle of attack, degrees (radians) 
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/3 angle of sideslip, degrees (radians) 

y flight path-angle, degrees (radians) 

e(t) error signal, volts 

9 pitch angle, degrees (radians) 

(p bank angle, degrees (radians) 

Symbols used in appendix C are defined therein. 

DESCRIPTION OF THE PANELS 


On the ACDS panel (fig. 1) the vertical -sc ale instruments are grouped so that 
short-period variables are to the left of the attitude indicator and the more slowly 
varying parameters are to the right. All the instruments have a common horizontal 



Figure 1.— ACDS instrument panel. 


Dynamic pressure 
Normal acceleration 
Angle of attack 
Pitch vernier (cross pointer) 
Attitude indicator 
Altitude rate 
Total velocity 
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read-line. On the standard panel (fig. 2) more or less conventional instruments are 
grouped so that the more important instruments are nearer the attitude indicator. 

X-15 pilots have become so thoroughly familiar with this layout and the instruments 
that they are able to garner a great deal of information from pointer rates and positions 
without having to "read" parametric values. 




B |B Q Angle of attack 

IB 


0 Normal acceleration 
3j Pitch vernier 
4J Attitude indicator 
Dynamic pressure 
Qn Inertial altitude 
7J Total velocity 
8) Altitude rate 


E- 12534 

Figure 2.— Operational instrument panel. 


EXPERIMENTAL METHOD 


Apparatus used for the experiment included the Flight Research Center's six- 
degree -of -freedom X-15 analog simulator and one additional computer console, which 
was programed to provide appropriate functions of error signals for analysis. A 
detailed description of the X-15 simulator is presented in reference 4. 

Sixteen subjects took part in the experiment: six current X-15 pilots, two Flight 
Research Center research pilots, four X-15 flight planners with considerable simu- 
lator experience, two research engineers with military piloting experience, and two 
engineers with limited private flying experience. 
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According to the originally proposed experimental design (fig. 3(a)), the 16 sub- 
jects were to be divided into eight matched pairs, each member of every pair 
assigned by coin -toss to one of the panels, and performance of the two resultant 

groups compared. This was a 
constrained randomization in that 
it was desired to have the same 
number of X-15 pilots, research 
pilots, engineers, and military 
pilots in each group. Strictly 
random assignment might easily 
have led to the objectionable 
result of having two of the six 
X-15 pilots in one group, four in 
the other, or perhaps even all six 
in one group. Since each of the 
16 subjects actually "flew" both 
panels to allow subjective com- 
parison, it was possible to collect 
enough data so that analysis of a 
second arrangement (fig. 3(b)) 
could be made as a check. Al- 
though this represents a proce- 
dural violation in experimental 
design, the results of this second 
test are reported since they act as 
an indication that inequalities in 
experience and natural ability be- 
tween groups were relatively 
minor. The second arrangement 
can be considered a replication 
of the experiment, in that the 
result of the coin-toss which 
determined each original subject - 
to -panel assignment could be 
assumed to be reversed. 

Mission flight plans were drawn up on the basis of actual flight experience and 
represented the two main categories of X-15 research missions, the "altitude" flight 
(mission 1) and the "heating" flight (mission 2). These flight plans, which are repro- 
duced in appendix A, were used in the same way as the standard flight request. 

Subjects practiced each mission using the flight plan until they had become thoroughly 
familiar with the various target parameter values and the sequence and timing of 
cross-checks. The piloting tasks are unique to the X-15 in that parameter ranges are 
greater and the flight regime varies more drastically than for any other operational 
aircraft. 

For the first mission, immediately after launch at an altitude of approximately 
45, 000 feet (13,700 meters), the engine was ignited and the throttle advanced to 
100 -percent thrust. The subject executed a pullup to an angle of attack of 11° (0. 19 rad) 
and held the value for approximately 30 seconds until the aircraft had rotated to a 
pitch angle of 39° (0. 68 rad). A vernier pointer (see figs. 1 and 2) showed null 


Group 1 

Group 2 

G 1 

G 2 

G 1 

G 2 

G 1 

G 2 

ACDS 

Standard 

a l 

*2 

A, display 

(a) Arrangem 

Group 2 

Group 1 

g 2 

G 1 

G 2 

G 1 

G 2 

G 1 

ACDS 

Standard 

a l 

*2 


B, experience 
(data runs) 


B, experience 
(data runs) 


A, display 

(b) Arrangement 2. 

Figure 3.— Experimental designs. 
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indication at a pitch angle of 39° (0. 68 rad), and the piloting task from this point until 
engine shutdown (at 5150 ft/sec (1570 m/sec) after about 80 seconds at full thrust) 
consisted of maintaining precise null indication, making correction for thrust mis- 
alinements and other disturbances while holding a wings -level (<p = 0°) (0 rad) attitude. 
This task involved control over a relatively short -period parameter. After shutdown, 
the aircraft coasted in a ballistic trajectory to the planned maximum altitude of 
250,000 feet (76,200 meters). 

The second mission differed in several respects. The subject rotated to a pitch 
angle of only 20° (0. 35 rad) and executed a push-over maneuver to Og after 42 seconds. 
At 4700 ft/sec (1430 m/sec) (after about 72 seconds) he reduced throttle setting to 
minimum thrust then extended the speed brakes to keep from exceeding airplane 
limits on dynamic pressure. If the subject followed the flight plan closely, he should 
have arrived at 75,000 feet (22,800 meters) altitude at the end of 74 seconds. His 
task from this point consisted of maintaining this altitude by steadily increasing angle 
of attack, meanwhile holding cp = 0° (0 rad) until burnout. Control was exercised 
over this rather slowly varying parameter through the use of g and h information, 
which required considerable skill and experience. 

When a subject felt he had achieved a reasonable level of proficiency and the 
data-processing circuits had been checked out, he made four runs for each mission: 
three for data, and one a "failure" run. During the failure run, both aircraft 
generators were failed, which required the subject to reset first the generators then 
the stability augmentation system (SAS). The position of the failure run in the four- 
run sequence was varied, and subjects were not told how many runs they would make. 
Opinions of the subjects were solicited to determine possible comparative effects of 
such distractions. Each subject was given a copy of a questionnaire immediately after 
his last data run and asked to complete it as soon as practicable. This questionnaire 
(see appendix B) provided subjects with an added feeling of participation and made it 
possible to compare subjective opinions with the results of objective performance 
measures. 

The parameters chosen as dependent variables, six for each mission, are listed 
in table I. 


TABLE I. - DEPENDENT VARIABLES 


Mission 1 

Mission 2 

1. Integral absolute error in <p 

2. Variance of absolute error in (p 

3. Integral absolute error in 0 

4. Variance of absolute error in 0 

5. Shutdown velocity error, 

absolute 

6. Peak altitude error, absolute 

1. Integral absolute error in (p 

2. Variance of absolute error in <p 

3. Integral absolute error in h 

4. Variance of absolute error in h 

5. Velocity error at reduction to 

minimum thrust, absolute 

6. Burnout altitude error, absolute 
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For an error signal e(t) over an interval of time t^ ^ t § t2 


integral absolute error = 



dt 


and 


variance of absolute error 


1 _ 




These quantities were calculated directly by the computer and recorded as they were 
generated. 

The precision of these measures is difficult to define, largely because of the 
characteristics of analog computation. Experimental error due to limitations 
regarding repeatability and accuracy were assumed to be random, so that only 
systematic differences would be revealed in statistical analysis. The fact that the 
X-15 simulator has been used successfully for several years in flight planning, pilot 
training, and handling-qualities investigations testifies to its reliability and accuracy 
for studies of this type. Only relative sizes of integral error scores were important. 
Thus, it was necessary only to adjust scaling of error signals experimentally to 
levels such that overloads in the equipment would not normally occur and "average" 
performance would result in easily measurable accumulated totals of at least several 
volts. Values of such parameters as shutdown velocity and altitude and peak altitude 
were read from digital voltmeters and compared with readings taken by an observer 
stationed at the cockpit. These recordings were, of course, only as accurate as the 
simulation itself. 

Error scores were transferred from the strip-chart records to data-matrix forms 
such as that shown in appendix C. From this form the necessary quantities could be 
readily calculated and used to derive the "F"-ratios which determine whether main and 
interaction effects are statistically significant. Performance differences were 
regarded as "consistently significant" when analysis of both arrangements (fig. 3) 
revealed significant F-ratios. This process is the two-dimensional factorial analysis 
of variance outlined in appendix C. A more complete discussion of the method is 
presented in reference 5. 

Since several parameters, notably integral absolute error in roll angle, showed 
such large variations in magnitude from subject to subject, thus weakening the power 
of the test, a nonparametric or "distribution-free" test was applied as a check. This 
test procedure is outlined in appendix C and discussed in detail in reference 6. 

Finally, performance scores were arranged in a "treatments x subjects" 
design (fig. 4) so that the scores (averaged over the three trials) of the entire group of 
16 subjects could be compared for the two panels, each subject acting as his own 

Variance can aid in discrimination between small continuous errors and large 
corrected errors, where integral error scores would not. Large integral error- 
score differences due, for example, to instrument bias (constant error) can thus be 
recognized, where suspected, by comparison with corresponding variance values. 
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"control. " A "t"-test of the difference in mean scores was made according to the 
procedure outlined in reference 7, pp. 90-93. Arrangement of the raw data in this form 
permitted direct examination of differences in each subject's performance on each 
panel, as well as of differences between experienced subjects and inexperienced 
subjects. 

Treatments 


Subjects 


ACDS Standard 



The simulator evaluation was followed by operational use of the ACDS panel in the 
X-15 airplane. After five flights by one of the pilots who had participated in the 
simulator evaluation, pilot opinion was solicited in order to obtain at least 
qualitative substantiation of the simulator results. 

RESULTS AND DISCUSSION 


It was anticipated from the beginning of the evaluation that results obtained would 
be limited to indications of differences between display panels and consequent 
suggested directions for further research. Also, it was expected that any differences 
noted would most likely be few and largely insignificant because of the relative 
uncertainty of typical measures of human performance. 

The following tentative interpretations and conclusions should be reviewed in the 
light of the limited preannounced goal of determining the suitability of the ACDS panel 
for flight as compared to the standard panel, the restrictions imposed by subject and 
equipment scheduling problems, and, above all, the difficulties attached to measuring 
human performance objectively on other than the most simplified of control-display 
tasks. 
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Table II summarizes the results of statistical analysis of the 24 performance 
measures. An F-ratio greater than 1. 44 is regarded as significant at the 0. 25 level of 
confidence. This means that differences as large as those observed would be expected 
to occur by chance only 25 percent of the time. The choice of the 0. 05 and 0. 01 levels 
of confidence in experimental work is frequently only a matter of convention. Under 
certain conditions the 0. 25 level may be more appropriate (in particular, when type 1 
errors (rejection of a "true” hypothesis) and type 2 errors (failure to reject a "false" 
hypothesis) are of equal importance, such as in exploratory work). Variance of 
integrated error was expected to be of use only as an indicator of the character of the 
error signal as generated. As such, it is not regarded as a direct performance 
measure. 


TABLE II. - SUMMARY OF F -RATIOS 


Mission 1 

Mission 2 

Dependent 

variable 

Panels 

Trials 

Trials 

X 

Dependent 

variable 

Panels 

Trials 

Trials 

X 




panels 




panels 

Arrangement 1 

cp integral 

0.45 

0. 33 

0.40 

<p integral 

0.74 

0.86 

3.33 

<p variance 

.23 

1. 15 

.06 

<p variance 

1.62 

. 18 

3.72 

9 integral 

.41 

.07 

.34 

h integral 

1.16 

.22 

1. 17 1 

9 variance 

.67 

.28 

.73 

h variance 

1.04 

1.12 

.62 

V error 

1. 94 

1.14 

.42 

V error 

a 3. 55 

.61 

1.90 

h error 

.53 

.25 

1.76 

h error 

a 2. 69 

.50 

.06 

Arrangement 2 

<p integral 

0 

.02 

.70 

(p integral 

1.02 

.50 

.98 

ip variance 

. 17 

.26 

.66 

<p variance 

2.07 

2.40 

1.74 

9 integral 

.02 

2.81 

1.75 

h integral 

1.91 

1.44 

2.41 

9 variance 

1. 18 

2.88 

1.94 

h variance 

.51 

.80 

.54 

V error 

.90 

.34 

.33 

V error 

a 2. 58 

.13 

1.25 

h error 

2.84 

.11 

.48 

h error 

a 2. 28 

.99 

1.46 


a F -ratios regarded as "consistently significant." 


Significant Differences 

Of the direct performance measures taken, only two^ showed consistent significant 
differences (i. e. , F-ratio greater than 1. 44 for both arrangements): absolute error in 
velocity at reduction to mi nim um thrust (target V = 4700 ft/sec (1430 m/sec)) and 
burnout altitude (target h = 75,000 ft (22,800 m)). Both measures were taken on the 
second mission. In both cases, results favored the ACDS panel. Examination of the 
altitude and velocity indicators on both panels shows that there is at least one good 

-^Though (p variance is not a direct performance measure, the F-ratios do indicate 
systematic differences in the way errors were generated, though not in the error scores 
themselves. This effect should be the subject of further investigation. 
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reason for these differences. In both instances, the scale resolution of the vertical- 
scale instruments was effectively doubled over that of the dial -types. The scale was 
expanded and there were more scale divisions (100 ft/sec per scale division versus 
200 ft./sec for the velocity indicator and 1000 feet per scale division versus 2000 feet 
for the altitude indicator). Whether or not the error score differences are caused by 
this factor alone is impossible to say, but these particular vertical -scale instruments 
are demonstrably superior for the particular tasks of reducing thrust at a pre- 
determined velocity and for holding a precise target altitude until burnout. 


Primary Control Task 

The measures of performance for prime control parameters, pitch-angle integral 
error for mission 1 and altitude integral error for mission 2, failed to show con- 
sistent significant differences (see table II). This can be explained by two factors: 

(1) pitch-angle deviation from the command value was displayed on both panels by means 
of a moving pointer on the attitude indicator, and (2) altitude -error measurement was 
sensitive only to fairly significant deviations, and performance appeared to be more a 
function of skill and experience than of display. Performance in the attainment of a 
particular altitude depends greatly on the use of instruments other than the altimeter, 
in particular, the g meter and rate-of-climb indicator. Despite rather great dif- 
ferences, between panels, in these three instruments, use of the combination ap- 
parently had no significant differential effect on performance. 

The fact that there are almost no consistently significant F-ratios in either the 
Trials or Trials x panels columns of table II indicates that subjects* performance did 
not improve significantly with runs and that performance with regard to runs was not a 
function of the panel being "flown" (i. e. , practice with the ACDS panel had been at 
least adequate for the comparison). 


Secondary Control Task 

Since roll angle was not a primary control parameter (the control requirement being 
merely to maintain cp = 0° (0 rad) as nearly as possible during the powered phase), but 
could be regarded as a secondary or loading task, it was hoped that error score differ- 
ences due to differences in workload between panels would be detected. Roll error was 
not expected to be a direct function of display, inasmuch as both panels incorporate the 
same all-attitude indicator, but differences might suggest that one panel or the other 
reduced workload on the primary tasks, thus allowing subjects to devote more time to 
the secondary task. Roll -angle -error integral failed to show significant differences 
under analysis by a parametric statistical test. But, since score magnitudes varied 
so greatly among subjects (from less than 0. 1 volt to as much as 80 volts), a non- 
parametric test was applied. Once again, no consistent significant differences were 
noted. However, since the tests of significance do not allow conclusions to the effect 
that there are no differences, it can be concluded only that if differences in workload 
level did exist the experimental measurements were not sensitive enough to detect them. 


Choice of Missions 

Mission 2 appears to have been a good choice for the program because of the 
relatively high-level workload it imposed on subjects; mission 1 seems to have been a 
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rather poor choice. This fact was demonstrated by one of the more experienced X-15 
pilots who flew this mission easily and successfully using only the three-axis attitude 
indicator, the angle-of-attack indicator, and the mission timer. A more challenging 
mission could have been designed especially for the evaluation, but representative, 
realistic missions seemed to be a more logical choice inasmuch as they would permit 
comparison with later actual missions. 


Treatments x Subjects Analysis 

The results of the "treatments x subjects" analysis are summarized in table III in 
terms of "t "-ratios. A t-ratio of 1. 75 indicates significance at the 0. 10 level of 
confidence. On the basis of these results, it would appear that the analysis leads to 
generally the same conclusions as drawn from the initial analysis, with perhaps 
greater reliability. Examination of raw scores revealed no apparent trends with 
respect to differences between experienced and inexperienced subjects. 


TABLE HI. - SUMMARY OF t -RATIOS 


Mission 1 

Mission 2 

Dependent 

variable 

t-ratio 

Dependent 

variable 

t-ratio 

(p integral 
<p variance 
9 integral 
9 variance 
V error 
h error 

0.44 
.03 
.32 
.89 
1.39 
a 3. 50 

(p integral 
<p variance 
h integral 
h variance 
V error 
h error 

1.34 
1.51 
1. 59 
1. 02 
a 2. 42 
a l. 96 


a t -ratios significant at the 0. 10 level of 
confidence. 


Questionnaire 

Data from questionnaires are so subject to misinterpretation that results can be 
reported fairly only in the form of summaries of replies to the more relevant queries. 
Appendix B presents a sample of the original questionnaire along with a tabulation of 
replies to most questions. Most participants indicated that they judged their own 
performance to be equivalent for the two panels. This opinion seems to bear out the 
results of the objective measures. 


FLIGHT SUBSTANTIATION 


After the evaluation was completed, the ACDS panel was flown successfully five 
times by the same pilot. Because of the necessarily limited number of flights, 
results of the evaluation are, of course, preliminary and consist mainly of pilot 
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opinion. There were no significant problems with respect to mechanical operation 
of the individual instruments. 

A questionnaire was given to the pilot who made the flights and comments were 
requested. The results are shown in appendix D. The pilot reported that his confi- 
dence in the ACDS panel increased with simulator practice and flight experience. 

After four flights, he began to prefer the tape instruments because they reduced 
cross-check time during the boost phase and generally provided greater resolution than 
the round-dial instruments. He reported, however, that he missed the "peripheral" 
information available from pointer rates and positions. 


CONCLUSIONS 


A comparative evaluation of the vertical -scale fixed-index (ACDS) instrument 
display panel and the "standard" X-15 panel resulted in the following conclusions: 

1. Missions can be carried out as accurately and successfully with the ACDS 
panel as with the "standard" model. Further training with the ACDS panel might, 
however, result in clearer distinctions in performance level. 

2. In determining particular target values, such as a definite shutdown velocity 
or a given altitude, the greater resolution of the vertical -scale instruments most 
likely resulted in significantly improved performance. 

3. For the most part, the dependent variables selected as performance measures 
proved to be either too sensitive to intersubject variability or not sufficiently direct 
functions of display to allow conclusions to be drawn about overall performance. 

4. Pilot opinion, as expressed in questionnaires, tends to support the objective 
results of the evaluation insofar as most subjects felt their performance and workload 
levels to be nearly equivalent for the two panels. 

Five flights of the ACDS panel in the X-15 airplane resulted in a limited 
operational substantiation of these conclusions on the basis of comments by the pilot. 


Flight Research Center 

National Aeronautics and Space Administration 
Edwards, Calif., January 11, 1967 
125-19-02-02-24 
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APPENDIX A 


ACDS PANEL EVALUATION FLIGHT PLANS 
FLIGHT PLAN NO. 1 - ALTITUDE 


Item t, sec ftxio 3 v * ft /see a, deg q, lb/ft 2 Event 


1 . 

0 

45 

790 

2 

145 

Launch, light engine, increase to 
100 percent thrust. Rotate to o^ll 3 , 
maintain 11° to 0 = 39 c . 

2. 

30 

54 

1800 

11 

550 

0=39°, maintain 0=39°. 

3. 

82 

139 

5150 

7 

50 

Shut down at 5150 ft /sec, push over to 
0°. Engine Master "OFF", RAS* 
"ON”, pitch SAS hi. 

4. 

170 

250 

4300 

0 

.8 

Peak altitude, extend speed brakes to 
20°. 

5. 

175 

— 


Special Instructions 

End of run. 


1. Use SAS 

Pitch off until burnout 
Roll - lo 
Yaw - hi 

2 . RAS off until burnout 

3. For ACDS panel, push ©, switch, and set a CO mmand = ^°* 

4. When generator failure occurs, 

a) reset both generators 

b) reset all SAS channels by switching to off, then to the proper gain. 


FLIGHT PLAN NO. 2 - HEATING 


Item t, sec ftxlO 3 V, ft /sec o, deg q, lb /ft 2 Event 


1 . 

0 

45 

790 

2 

145 

Launch, light engine, increase to 
100 percent thrust. Rotate to 10° ot 
until 0 = 20°. 

2. 

18 

47 

1500 

10 

450 

0 = 20°, maintain 0 = 20°. 

3. 

42 

59 

2700 

4 

870 

Push over to 0 g. 

4. 

72 

75 

4700 

0 

1200 

Reduce to minimum thrust. 

5. 

74 

75 

4800 

0 

1250 

Modulate speed brakes (~35°) to 
maintain slow longitudinal accel- 
eration, increase ot to maintain 
h = 75,000 feet. 

6 . 

90 

75 

5020 

3 

1400 

Burnout. Retract speed brakes, 
maintain h = 75, 000 feet. 

7. 

— 

— 

— 

— 

— 

h = 0, maintain h = 0. 

8. 

125 

— 

— 

Special Instructions 

End of run. 


1. Use SAS 

Pitch - hi 
Roll - lo 
Yaw - hi 

2. RAS off 

3. For ACDS panel, push 0, /3 switch, and set <* comman d = 10°» T = 0 • 

4. When generator failure occurs, 

a) reset both generators 

b) reset all SAS channels by switching to off, then to the proper gain. 


* Reaction augmentation system 



APPENDIX B 


SUMMARY OF REPLIES FROM 12 OF 16 SUBJECTS 
QUESTIONNAIRE - ACDS Panel Evaluation 


NAME: 

DATE: 


Key (items 5, 9, 16) 

O Research pilots 
□ Engineers 
O Flight planners 


For questions requiring a rating, place an "X" on the scale provided. Please feel free 
to add any comments you may want to make relative to any of the questions. 


1. Approximately how much instrument flight experience have you had ? 


From 0 to nearly 1200 hours simulated and actual flight time. Average, 
4-00 hours . 


2. What previous experience, if any, have you had with vertical-scale instruments 
of the type used in the ACDS panel ? 

2 - took part in earlier evaluation 

3 - took part in single-instrument evaluation or F-106 panel flight tests 
1 - took part in energy-management work using ACDS 

5 - no previous experience 

1 - did flight-planning work with ACDS panel 

3. Do you believe that vertical -scale fixed-reference (tape) displays have definite 
value in the X-15 mission ? 


7 ~ yes 
4 - no 

1 - "yes and no" 


4. Indicate your feeling about the suitability of the two missions for comparison 
and evaluation of the panels . 

6 - O.K. to good 
5 - O.K. with qualifications 
1 - no response 


Light 


5. How did pilot workload compare between panels ? 

90 D$Q | OO [ OO | 


Moderate 


Heavy 


g B, o o □ op co 


Light 


Moderate 


Heavy 


Standard 


ACDS 
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6. Did you feel as if you were forced to concentrate harder to pick out definite 
values of parameters (for example, in accomplishing shutdown at a definite velocity) 
for the ACDS panel ? 

4 - yes 

5 - no 

2 - yes, slightly 

1 - "yes and no" 

7. How did you feel about the naturalness and interpretability of vertical-scale 
instruments as compared to standard ones (for example, in regard to direction and rate 
of tape movement, scaling, and sensitivity) ? 


1 1 1 

More difficult 

1 1 1 

1 1 1 

Easier 

I I 1 

Vertical scale 

More difficult 

Easier 

Standard 


This scale largely misinterpreted; replies difficult to summarize. 

8. Do you think further training with the ACDS panel would improve your perform- 
ance significantly ? 


7 - yes 
5 - no 


9. How do you rate the overall suitability of the panels for the X-15 mission ? 

O | O I § op <3> Cp 2_ 


Less 


Less 


1 


9 & 


<a> 


e 


More 


More 


■ACDS 


Standard 


10. Does the fact that parameter values must be read on the ACDS panel bother 
you? That is, do you notice loss of cues you may be used to picking up from judgment 
of needle or pointer position and rates ? 


11 - yes 

0 - no 

1 - "yes and no" (depends on parameter) 

11. Do you think some sort of "hybrid" display panel, combining the best features 
of both ACDS and standard panels should be designed ? 


5 - yes (qualified) 

1 - no, stay with fixed scale, moving pointer 

2 - worth investigating 
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2 - have to try it to judge 
1 - no response 

1 - think better design possible 

12. Do you prefer having a (angle of attack) displayed on a moving-pointer or on 
a moving-tape display ? What about velocity and altitude — does a circular scale seem 
more legible, or do you prefer it because you are used to it ? 

1 - no response 

5 - prefer moving pointer, circular or linear scale 
5 - prefer tape 
1 - "mixed feelings" 


13. Immediately after emergency conditions, did you experience any difficulty or 
confusion upon returning your attention to either panel? 


5 - yes 
7 - no 


1 - ACDS better 

2 - standard better 

2 - no difference noted 


14. What parameters do you think would best be used to measure pilot performance 
in evaluation of displays ? 


Opinions vary widely; difficult to summarize. 


15. Do you think simulator results will be valid or do you think valid panel evalua- 
tion can be made only in flight ? That is , do you think the simulation has been tested 
and proved to the point that results of this type of evaluation can be trusted ? 


2 - yes 
2 - no 

8 - valid, subject to confirmation by flight test 

16. How much emphasis do you think should be placed on pilot judgment or opinion 
in displays evaluation ? 

I I [DO | □ p t OO 

Not much Some Great deal 
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STATISTICAL METHODS 
FACTORIAL ANALYSIS 


The model for the two-dimensional factorial analysis of variance with repeated 
measures on one factor (see ref. 5) is shown below: 


a I a 2 


t>2 B (experience) 
b 3 

ACDS Standard 

A (display) 


G 1 

G 2 

Gl 

g 2 

G 1 

G 2 


in which A represents the between-subject variable with levels ap i = 1, 2, . . . p 
(here, panels ap a£), B represents the within-subject variable with levels bj, 
j = 1, 2, . . . q (here, trials bp b 2 , and b 3 ). 

Subjects are denoted by the subscript 1 = 1, 2, . . . n (n = 8), and X ^ denotes 

the score for subject l in all i j of the data matrix. A sample data matrix form is 
shown below: 


Mission Profile No. Dep. Var. 


Display 

Subject 

Trial 1 

Trial 2 

Trial 3 

Total 


1 

2 

X U1 

X 121 

*131 


Standard 

3 

4 

5 

6 

— 

— 

— 



7 

8 

*118 

X 128 

x 138 



Total 






9 

10 

*211 

X 22l 

x 231 


ACDS 

11 

12 

13 

14 

— 

— 

— 



15 

16 

X 218 

X 228 

X 238 



Total 






Total 
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It is assumed that both groups (each G assigned to one panel) are random samples 
from the same population and that distribution of scores for each treatment condition 
population is normal and has the same variance. The null hypothesis that mean 
scores for each treatment condition population are equal is tested to determine the 
following: 

1. Differences due to any one variable (e. g. , panels) across all levels of the 
remaining variable (trials), 

2. Differences due to combinations of two (or three) variables across all levels 
of the remaining variable but not due to effects of the type described in item 1 (inter- 
active effects). 

The hypothesis is tested by means of the F-test. A confidence level of 0. 25, a 
commonly used value for experiments with few subjects and trials, was chosen for 
these tests. 


To determine the proper ratios for the F-test, the total sum of squares of devia- 
tion of the variables Xjjj from their sample mean X is first broken down into com- 
ponents as follows: 


izztai - x) 2 


between -subject variance 
*2 


-*) - *1). -*U+ X t.) 


within -subject variance 


where subscripted dots indicate summation over replaced indices, for example, 


Xt. . ‘ZZftjl- 

J 2 


Then it is merely necessary to divide each sum of squares by its 


degrees of freedom and form the proper ratio depending on the hypothesis to be tested. 
For example, in testing for differences due to panels, the first and second terms would 
form an F-ratio. If the computed value of this F should exceed the expected value of 
F in the appropriate distribution (determined by ^numerator* ^denominator ) for 


the chosen confidence level, the null hypothesis may be rejected. 


COMPUTATIONAL PROCEDURES 


To simplify the computations required in these tests, the following quantities are 
calculated first: 


r n q p \2 

wfZZZxyjj 

,2=1 j=l i=l 1 


n q p 

(2) Z Z Z X ijZ 2 
2=1 j=l i=l 
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P / n q 

(3) I I Z 
i=l V=1 j=l 


nq 



q A P \ 2 

np 


q p A 

( 5 ) II I 

3=1 i=iy=l 



n 


(3) Z Z 

1=1 i=l\ 



n 

Then, the following quantities are calculated: 


q 


SSbet _ (6) “ (1) 


SS A = (3) - (1) 
ss error(b) = < 6 > ~ < 3 ) 

SSwith = ( 3 ) " 

SS B = (4) - (1) 

SS AB = (5) - (3) “ (4) + (1) 

ss error(w) = ( 2 ) ~(5) -(6)+ (3) 

df bet = n - P “ 1 
df A = p - 1 

^errorlb) = P ( n " 1) 

d ^wiTH = n P (q “ 1) 

df B = q - 1 

df AB = (P - 1) (q - 1) 
^errorlw) = P< n ' *> ' X) 
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SS A 
MS A df A 

SS error (b) 
MSerror(b) = d£ error(b) 

ss B 

ms b - ^ 

A/r o SS AB 
ms ab df AB 

_ ^error(w) 
Ms error(w) = dl error(w) 


Differences in the measure attributed to each variable (A, B) and to inter- 

action (A x B) are tested as follows: 

1. Determine the desired level of confidence a. In this case, a = 0.25. 

2. If the value of F from the following table exceeds the value of F in the 
appropriate distribution (determined by dJnumerator’ ^denominator) for the chosen 
value of a, the null hypothesis that "There is no difference in Xyj attributable to 

this source" is rejected (i. e. , if the null hypothesis is true, the difference is so large 
that it would occur by chance less than the fraction a of the time). 


Source 

F 

^numerator 

^denominator 

A 

ms a 

^error(b) 

df A 

^error(b) 

B 

ms b 

MS error ( w ) 

df B 

^error(w) 

A x B 

ms ab 

M ^error(w) 

^AB 

^error(w) 
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DISTRIBUTION-FREE TEST 


Since several of the dependent variables appeared to violate the assumptions under- 
lying the parametric analysis of variance, a distribution -free or nonparametric test 
was used (see ref. 6). This test is based on the X 2 distribution, with the following 
notation: 

X^i = score for subject l, cell ij of the data matrix, as in the preceding analysis 
of variance design 

n^j = number of scores in cell ij (in this case, all n^ = n) 

q P q P 

n i = n ij > n . j _ n ij » ^ — n ij 

j=l J J i=l J j=l i=l J 

Find Md = median of the entire set of scores. If there is no such number, let 

Md be the number which most nearly divides the set X^ into two equal sets of 
numbers. 

Let 

afjj = number of X^ > Md 
bfjj = number of X^ < Md 
N a = total number of Xjj i > Md 


= total number of X^j i < Md 

q 

q 

II 

.M 

£ 

b£ i. = S My 


1=1 

p 

P 

•j 7 

II 

b V§ bf « 


Then calculate 


q P 

: t 2 =ZZ 

j=l i=l 


af. 


N / N. \ 2 

- n..-rr- (bf. . - n..— rr) 

ii l] N V 11 11 N / 


U ii 


U iL 


5l 

n ij. N 


n ij. N 
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Main and interaction effects are tested as follows: 

1. Select a level of significance a (0.25). 

2. Reject the hypothesis that a variable has no effect on the measure X^j, at ex 

level of significance, if the calculated X 2 exceeds the X 2 table value for the chosen 
ex and appropriate degrees of freedom (df) as indicated in the following table: 


Variable 

X 2 

df 

A 

*A 2 

P-1 

B 

X B 2 

q-1 

A x B 

*I 2 = V - V - *B 2 

(P - 1) (Q - 1) 
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RESULTS OF FLIGHT EVALUATION 
QUESTIONNAIRE - ACDS Panel Flight Evaluation 


NAME: 

DATE: 

1. Approximately how many hours have you spent in flight preparation with the 
ACDS panel ? 

150 hours 

2. Has your confidence in the ACDS panel increased with experience (since the 
simulator evaluation) ? 

Yes. For the first "JO to 80 hours on the simulator and the first two 
flights I would have preferred round instruments . By the third flight I was 
neutral, and after the fourth flight I preferred the tapes (with reservations). 

3. Have you noticed any gross difference between simulator and airplane confidence 
level (with the addition of motion cues as well as the need to monitor engine, subsystem 
instruments) ? 

Confidence level is the same. There is a definite reduction in cross- 
check during the flight. This is due to tighter check of the primary parameters 
(e.g., 0) leaving less time for checking lesser parameters. 

4. Do you believe, after flight experience, that vertical scale, fixed-reference 
displays have definite value in the X-15 mission? 

Yes. They unburden the pilot during boost by reducing cross-check time. 
Generally, they also provide better resolution (e.g., vertical speed). 

5. How does pilot workload compare (in flight) between panels ? 


— 1. 

1 1 


1 1 

Heavier 

1 

l \y 

A 

1 

Lighter 

1 l_ 

Heavier 

A 


Lighter 


6. How do you rate the overall suitability of the two panels for 


_J 1 

Less 

1 l 

J 1 1 V L- 

More 

1 1 V 1 

Less 

A More 


ACDS 

Standard 

flight ? 

ACDS 

Standard 
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7. Do you think the simulator evaluation was valid, after flight experience ? 

Results of the simulator effort indicated little measurable difference in performance 
between panels . 

Yes. Sounds like a valid evaluation. 

8. Did you experience any instances of momentary confusion or disorientation 
(other than that due to mechanical difficulties) ? 

Not in flight. I have experienced some confusion in the sign of angle- 
of- attack rate during pitch-damper-off work in the simulator. 

9. How has in-flight performance been in regard to planned versus achieved peak 
altitude, shutdown velocity, etc.? 

No profile errors have resulted from erroneous tape information. There 
does appear to be a slight lag of the velocity tape in flight. 

10a. What features, if any, of the ACDS panel do you like ? 

Resolution of all tapes, range of h tape. 

10b. What features do you find objectionable, if any ? 

Can misinterpret direction of angle-of- attack rate of change. Periph- 
eral information is not as good as on round dials. Normal accelerometer is not 
as reliable as mechanical one. 
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